1. Field of the Invention
The present invention relates to an image processing apparatus and an image processing method.
Priority is claimed on Japanese Patent Application No. 2010-172157, filed Jul. 30, 2010, the content of which is incorporated herein by reference.
2. Description of the Related Art
All patents, patent applications, patent publications, scientific articles, and the like, which will hereinafter be cited or identified in the present application, will hereby be incorporated by reference in their entirety in order to describe more fully the state of the art to which the present invention pertains.
In an image processing apparatus such as a still-image camera, a moving-image camera, a medical endoscope camera, or an industrial endoscope camera, for example, a plurality of image processes are performed when one still image is generated. Japanese Patent Publication No. 4179701 discloses an image processing apparatus for performing a plurality of image processes within the image processing apparatus using little memory unit access.
FIG. 5 is a block diagram illustrating a schematic configuration of an image processing apparatus in accordance with the related art. Under control of a CPU 2 that controls each unit connected to a bus 1, the image processing apparatus in accordance with the related art shown in FIG. 5 pre-processes an imaging signal output from a solid-state imaging apparatus such as CCD by a pre-processing circuit 3. The image data pre-processed by the pre-processing circuit 3 is temporarily stored, for example, in a frame memory unit 4 such as a dynamic random access memory (DRAM), via the bus 1. Subsequently, the image process processing unit 5 reads the image data stored in the frame memory unit 4 via the bus 1. After the image process processing unit 5 performs a plurality of image processes on the read image data, their result is output to a Joint Photographic Experts Group (JPEG) processing circuit 6. The image data JPEG-processed by the JPEG processing circuit 6 is re-stored in the frame memory unit 4 via the bus 1.
When each image process is performed in the image processing apparatus in accordance with the related art as described above, access to the frame memory unit 4 via the bus 1 is performed, for example, by direct memory access (DMA). In the image processing apparatus in accordance with the related art shown in FIG. 5, image process circuits (image process circuits 5-1 to 5-n) within the image process processing unit 5 are connected in series to implement the plurality of image processes in a small DMA area by pipeline-processing the plurality of image processes.
Incidentally, a distortion aberration generally occurs in an optical system of a camera. FIGS. 6A to 6C are diagrams illustrating examples of distortion aberrations of an optical system. For example, if an image processing apparatus captures an image of a grid subject as shown in FIG. 6A, the captured image has a barrel distortion as shown in FIG. 6B or a pincushion distortion as shown in FIG. 6C due to the distortion aberration of the optical system.
In recent cameras, optical systems are usually equipped with an optical zoom function, and distortion aberration states in the wide angle side and the tele side such as the barrel distortion appearing in the wide angle side and the pincushion distortion in the tele side are usually varied. Japanese Unexamined Patent Application, First Publication No. 2005-45514 discloses an image processing apparatus including a distortion correcting circuit for correcting a distortion aberration varying with the state of the optical system as the image process circuit.
In the image processing apparatus in accordance with the related art disclosed in Japanese Unexamined Patent Application, First Publication No. 2005-45514, the case where the distortion correcting circuit for correcting the distortion aberration is included, for example, as the image process circuit 5-2 within the image process processing unit 5 of the image processing apparatus of the related art shown in FIG. 5 is disclosed. In the image processing apparatus in accordance with the related art shown in FIG. 5, the JPEG processing circuit 6 performs JPEG processing after image processes are performed by the image process processing unit 5. Because the JPEG processing is performed in units of 8-pixel image data in a horizontal direction, the image process processing unit 5 initially performs aspect conversion on the image data, and outputs the image data to the JPEG processing circuit 6 after performing a plurality of processes for the image data subjected to the aspect conversion. Thus, in the image process circuit 5-2, which is the distortion correcting circuit in accordance with the related art, an aspect conversion circuit (the image process circuit 5-1), which is a preceding-stage image process circuit, acquires the image data temporarily stored in the frame memory unit 4 by DMA, and the image data aspect-converted by the aspect conversion circuit is input as input image data. The distortion correcting circuit in accordance with the related art corrects the distortion aberration of the optical system for the input aspect-converted image data.
Here, the aspect conversion circuit (the image process circuit 5-1), which is a preceding-stage image process circuit that inputs input image data to the distortion correcting circuit of the related art, will be described. FIG. 7 is a diagram schematically illustrating an operation in which the aspect conversion circuit reads image data from the frame memory unit 4 and outputs input image data to the distortion correcting circuit. FIG. 7 shows one-screen image data stored in the frame memory unit 4. The aspect conversion circuit divides the one-screen image data stored in the frame memory unit 4 into areas (hereinafter referred to as block lines) 41 each having a width of the predetermined number of columns. The width of the number of columns of each block line 41 is set so that the area of each block line 41 overlaps that of an adjacent block line 41 with respect to each column-direction area.
The aspect conversion circuit reads image data within the block line 41 in a column direction of the frame memory unit 4 like a read direction 42 shown in FIG. 7. Thereafter, the aspect conversion circuit outputs the read image data of the block line 41 in a row direction of the frame memory unit 4 like an output direction 43 shown in FIG. 7 perpendicular to the column direction. The image data output in the row direction of the frame memory unit 4 like the output direction 43 is input to the distortion correcting circuit as input image data. As described above, the aspect conversion circuit inputs the aspect-converted image data to the distortion correcting circuit. One row (line) of the image data read in the output direction 43 shown in FIG. 7 is referred to as a “unit line.”
FIGS. 8A and 8B are diagrams illustrating a schematic configuration and a schematic operation of the aspect conversion circuit. The aspect conversion circuit as shown in FIG. 8A has two small memory units (memory units 511 and 512). Switching control is performed by a control unit (not shown) for the memory units 511 and 512 so that when image data read from the frame memory unit 4 is written to one memory unit, image data written to the other memory unit is read and output to the distortion correcting circuit as shown in FIG. 8B. FIG. 8A shows an example of a state in which image data read from the frame memory unit 4 is written to the memory unit 511 and image data from the memory unit 512 is read and output (read) to the distortion correcting circuit.
When the image data read from the frame memory unit 4 is written to the memory unit 511 or 512, the image data is scanned (raster-scanned) in the column direction of the frame memory unit like the read direction 42 shown in FIG. 7, and divided into, and written to, the block lines 41. The memory unit 511 and the memory unit 512 are, for example, memory units capable of storing data having a burst length of the frame memory unit 4, not memory units capable of storing data having the same length as in the column direction of the frame memory unit 4. Accordingly, the memory unit 511 and the memory unit 512 are memory units having a capacity of (Width of Predetermined Number of Columns×Burst Length).
When the image data stored in the memory unit 511 or 512 is read (output) to the distortion correcting circuit, the image data is read and output while being scanned in the row direction of the frame memory unit 4 like the output direction 43 shown in FIG. 7. As described above, the image data is aspect-converted by changing a direction of writing to the memory unit and a direction of reading from the memory unit.
Next, the distortion correcting circuit (the image process circuit 5-2) that corrects a distortion aberration in the image processing apparatus of the related art will be described. FIG. 9 is a block diagram illustrating a schematic configuration of the distortion correcting circuit in accordance with the related art. A distortion correcting circuit 52 in accordance with the related art shown in FIG. 9 corrects input image data input from the aspect conversion circuit, and outputs output image data after correction to a next-stage image process circuit.
As shown in FIG. 9, the distortion correcting circuit 52 includes an interpolation position generating unit 521, a distortion correction coordinate transforming unit 522, a line memory unit 523, a memory control unit 524, and an interpolation calculating unit 525. The interpolation position generating unit 521 generates an interpolation position indicating a position of output image data to be output after an interpolation. The distortion correction coordinate transforming unit 522 transforms the interpolation position generated by the interpolation position generating unit 521 into coordinates indicating a position of input image data before a distortion correction by applying a distortion correcting equation of a distortion aberration characteristic corresponding to a preset zoom position of the optical system or the like. Thereby, it is possible to obtain the position (coordinates) of input image data to be used for an interpolation to be executed for data of the interpolation position of the output image data generated by the interpolation position generating unit 521.
The line memory unit 523 temporarily stores input image data input from the image process circuit 5-1 for each unit line. Also, the line memory unit 523 includes storage areas for a predetermined plurality of unit lines, and has a form in which the storage areas are connected in a ring shape. It is possible to store input image data of a larger number of unit lines than the predetermined number of unit lines by subsequently rewriting input image data of unit lines unused in an interpolation calculation.
The memory control unit 524 controls the line memory unit 523 based on the coordinates of the input image data before the distortion correction into which the transformation is performed by the distortion correction coordinate transforming unit 522. The interpolation calculating unit 525 generates the output image data by carrying out the interpolation calculation on the input image data stored in the line memory unit 523 based on the coordinates of the input image data before the distortion correction into which the transformation is performed by the distortion correction coordinate transforming unit 522, and outputs the output image data to a next-stage image process circuit.
Here, the interpolation calculation of the distortion correcting circuit 52 in accordance with the related art will be described. If an interpolation calculation of predetermined output image data is performed, the interpolation calculating unit 525 generates output image data, for example, by a bilinear interpolation or a bicubic interpolation, using input image data of a row (line) including input image data before a distortion correction and input image data of the previous or next line including input image data located at its periphery. FIG. 10 is a diagram schematically illustrating an example of the line memory unit provided in the distortion correcting circuit 52 and input image data stored in the line memory unit in accordance with the related art. FIG. 10 shows a storage area capable of storing input image data for 16 lines (unit lines) included as the line memory unit 523, a line of input image data before a distortion correction corresponding to output image data to be interpolated when output image data of one row (line) is output, and a range (the number of unit lines) of input image data necessary for an interpolation calculation. In the example shown in FIG. 10, an example in which output image data is generated by the bicubic interpolation using a line where input image data before a distortion correction corresponding to the output image data is stored, input image data of its previous line, and input image data for up to two subsequent lines.
Incidentally, a distortion shape of each line of an image captured using an optical system having a distortion aberration as shown in FIG. 6 differs according to each line. Thus, the number of lines (the number of unit lines) of input image data necessary for outputting output image data of a predetermined line by the interpolation calculating unit 525 also differs according to each line for outputting the output image data. For example, as shown in FIG. 10, the necessary number of lines (number of unit lines) of input image data differs according to each row (line) from which the output image data is output so that input image data of first to seventh unit lines is necessary if output image data of one row (line) is generated based on input image data shown on a line a and input image data of ninth to thirteenth unit lines is necessary if output image data of one row (line) is generated based on input image data shown on a line b.
However, it is difficult to pre-obtain the number of lines (the number of unit lines) of input image data necessary for outputting output image data of a predetermined line, and it is not possible to obtain the appropriate number of lines of input image data necessary for the interpolation calculation if coordinate transformation is not actually executed. If interpolation processing is performed without obtaining the necessary number of lines of input image data by actually executing the coordinate transformation, for example, if the number of lines of input image data stored in the line memory unit 523 is less than the number of lines of input image data necessary for the interpolation calculation, the processing of the interpolation calculation becomes erroneous without being normally completed. If the processing of the interpolation calculation becomes erroneous, it is not possible to efficiently perform interpolation processing because the interpolation processing is re-executed after the number of lines of input image data stored in the line memory unit 523 is greater than or equal to the necessary number of lines.
Because of this, it is possible to increase the number of lines of input image data stored in the line memory unit 523, that is, the prepared number of unit lines capable of being temporarily stored in the line memory unit 523, as a countermeasure for normally completing the interpolation processing without being affected by a distortion shape of each line. However, because input image data of a unit line unused in actual interpolation processing is also stored, the use efficiency of the line memory unit is degraded. Because it is necessary to prepare more line memory units, this becomes a factor that increases a circuit scale of the distortion correcting circuit.
Japanese Unexamined Patent Application, First Publication No. 2005-45514 also discloses a distortion correcting circuit including a distortion correction range calculating unit that executes a range of input image data before a distortion correction for a range of output image data to be output after an interpolation before the distortion correction is processed. In order to distinguish from the distortion correcting circuit 52 that does not include the distortion correction range calculating unit, the distortion correcting circuit of the related art including the distortion correction range calculating unit is referred to as a “distortion correcting circuit 53.” The distortion correcting circuit 53 pre-obtains coordinates of four sides of the input image data before the distortion correction by carrying out the same distortion coordinate transformation calculation as that of the distortion correction coordinate transforming unit 522 along a range of output image data, for example, four sides of the block line 41. This is based on an idea that it is possible to avoid an error in the processing of the interpolation calculation because a range of input image data necessary for carrying out an interpolation calculation of all output image data is input image data existing within a pre-calculated range by pre-calculating a range (coordinates) of input image data necessary for generating output image data of the outermost periphery in a range in which the output image data is output.
In Japanese Unexamined Patent Application, First Publication No. 2005-45514, the number of unit lines of a front side and the number of unit lines of a rear side to be used when the interpolation calculation is carried out are decided by assuming that a skew (distortion) of one end of left and right ends within the four sides of the range of the input image data obtained is a maximum skew (distortion) to be generally considered. That is, a range (the number of unit lines) of input image data to be used in the interpolation calculation is decided based on input image data of any one end having a large distortion between the left and right ends of the block line 41. The number of unit lines of the front side and the number of unit lines of the rear side decided here are uniformly set as the number of unit lines to be used when the interpolation calculation of the block line 41 is carried out.
FIG. 11 is a diagram illustrating a method of obtaining the number of unit lines of a front side and the number of unit lines of a rear side to be used when an interpolation calculation is carried out in the distortion correcting circuit 53. FIG. 11 shows a position of input image data of any one end of left and right ends of a range of input image data obtained by the distortion correction range calculating unit. When the number of unit lines of the front side and the number of unit lines of the rear side to be used in the interpolation calculation are obtained, a position (coordinates) of input image data A (an upper end of an input image data line in FIG. 11) for which an interpolation is initially performed is first designated as the center.
Based on a difference between the position (coordinates) of the input image data A serving as the center on the line of the input image data and a position (coordinates) of input image data H maximally separated on the front side (the left side in FIG. 11), the number of unit lines of the front side including the input image data H previous to the input image data A serving as the center is designated as a post-unit line buffer (ULB). Based on a difference between the position (coordinates) of the input image data A serving as the center on the line of the input image data and a position (coordinates) of input image data C maximally separated on the rear side (the right side in FIG. 11), the number of unit lines of the rear side including the input image data C after the input image data A serving as the center is designated as a pre-ULB. As described above, the number of unit lines of the front side (the post-ULB) and the number of unit lines of the rear side (the pre-ULB) are respectively obtained for lines of the input image data of the left and right ends of the input image data range.
One larger post-ULB value (the number of unit lines) between a post-ULB value obtained from a line of the left end of the input image data range and a post-ULB value obtained from a line of the right end of the input image data range is designated as a final post-ULB. One larger pre-ULB value (the number of unit lines) between a pre-ULB value obtained from the line of the left end of the input image data range and a pre-ULB value obtained from the line of the right end of the input image data range is designated as a final pre-ULB. The post-ULB and the pre-ULB are uniformly set as the number of unit lines to be used when the interpolation calculation is carried out.
For example, the distortion correcting circuit 53 sets the post-ULB and pre-ULB in the memory control unit 524 of the distortion correcting circuit 52 of the related art shown in FIG. 9, and controls the line memory unit 523 based on coordinates of image data before a distortion correction transformed by the distortion correction coordinate transforming unit 522 and the set post-ULB and pre-ULB values. That is, the memory control unit 524 controls the number of lines stored in the line memory unit 523 so that the number of lines of input image data stored in the line memory unit 523 satisfies the set post-ULB and pre-ULB. Thereby, the distortion correcting circuit 53 disclosed in Patent Document 2 is configured to avoid an error in the processing of an interpolation calculation and efficiently carry out the interpolation calculation.
However, it is not always true that a skew (distortion) at a left end or a right end of four sides of an input image data range is maximized in a distortion shape of each line in an image captured by an optical system having a distortion aberration. FIG. 12 is a diagram schematically illustrating an example of a distortion shape of a line of an input image data range. For example, there is also the case where a skew (distortion) of a left-end line L or a right-end line R is not maximized and a skew (distortion) of a center line C is maximized as in the example shown in FIG. 12. For example, if the skew (distortion) of the center line C is greater than in the set post-ULB and pre-ULB, the processing of an interpolation calculation becomes erroneous, interpolation processing is re-executed after the number of lines of the input image data stored in the line memory unit 523 is greater than or equal to the necessary number of lines, and the interpolation processing is not efficiently performed.
In consideration of the above-described case, it is possible to set values having margins in the post-ULB and the pre-ULB even in the distortion correcting circuit 53 as disclosed in Japanese Unexamined Patent Application, First Publication No. 2005-45514. However, if the post-ULB and pre-ULB having the margins are set, line memory units of which the number is greater than the number of lines to be used in actual interpolation processing are used. The distortion correcting circuit 53 can set margins to be small as compared with the distortion correcting circuit 52 because the post-ULB and the pre-ULB are pre-set, but the efficiency of line memory use is degraded because it is difficult to optimize margin amounts. Because it is necessary to prepare many line memory units considering the margins, a circuit scale of the distortion correcting circuit is increased.
If the efficiency of line memory use is degraded, a phenomenon that each image process to be pipeline-processed in an image processing apparatus is stopped (stalled) easily occurs. If the pipeline processing is actually stalled, it is necessary to re-execute each image process to be pipeline-processed and the entire processing time of the image processing apparatus becomes long. Alternatively, the entire processing of the image processing apparatus may not be normally completed. As a method of preventing the pipeline processing from being stalled in the image processing apparatus, it is possible to prepare a line memory unit in each image process circuit. However, in this case, a circuit scale of the image processing apparatus is further increased.